Volume 1

This research focuses on multimodal pedagogical communication in synthetic (virtual) worlds. The study investigates the interplay between verbal and nonverbal interaction which supports verbal participation and production in a foreign language. This is analysed from a socio-semiotic perspective of multimodality within the context of a course held in the synthetic world Second Life, which adopted a Content and Language Integrated Learning (CLIL) approach. The course, entitled 'Building Fragile Spaces', formed part of the European project ARCHI21. It was designed for higher education students of Architecture whose foreign language was either French or English. The interplay between verbal and nonverbal interaction is examined from three different angles. Firstly, considering the opportunities synthetic worlds offer for the co-creation of the environment through building activities and for collaboration, the role played by the nonverbal mode during a collaborative building activity is investigated. The study questions whether the use of the nonverbal mode impacts on the participation of students in the verbal mode, and whether any interplay exists between these two modes that influences verbal production. Secondly, use of the nonverbal mode by students in inworld identity construction is addressed and considered with reference to their verbal interaction, and participation, in the foreign language. Thirdly, the research concentrates upon interplay between the audio and textchat modalities in the verbal mode. More specifically, the focus is on whether the textchat plays a role during interaction, considering it is in competition not only with the audio modality but also with several nonverbal modalities; and on whether the textchat modality can serve for feedback provision on language form. This thesis seeks to contribute to the methodological considerations to allow research to move beyond speculative and anecdotal examples of multimodal pedagogical communication in synthetic worlds. A typology of nonverbal and verbal modalities is proposed, and then drawn upon, to extend a previous methodology suggested for multimodal transcription to interactions in synthetic worlds. Considering, within the fields of Social Sciences and Language Sciences, the more general research problem to render research data used for analyses visible and publically accessible, the study adopts a LEarning and TEaching Corpus (LETEC) methodological approach. Constituting a structured corpus allows for contextual analyses of the data collected during the 'Building Fragile Spaces' course. This research offers insights into how verbal participation increases with reference to the proxemic organisation of students, the customization of students' avatar appearance and an increased use of nonverbal acts. Concerning verbal production, the study shows how avatar movement in the nonverbal mode was used as a strategy to overcome verbal miscommunication when expressing direction and orientation and also the benefits of using the textchat modality for feedback on language form in order to support learners’ productions in the audio modality. In light of these results, the study suggests some considerations concerning the design of pedagogical activities for language learning within synthetic worlds.